{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model: AdaBoost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import precision_score, recall_score, accuracy_score, f1_score, confusion_matrix, classification_report\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "# Importing the model\n",
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "from sklearn.externals import joblib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading in Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_excel('../top10_corr_features.xlsx')\n",
    "df = df.drop(df.columns[0], axis = 1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scaling the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# StandardScaler: 可保存训练集中的均值、方差参数，然后直接用于转换测试集数据。\n",
    "scaler = StandardScaler()\n",
    "\n",
    "features_df = df.drop([\"Decision\"], 1)\n",
    "\n",
    "scaled_df = pd.DataFrame(scaler.fit_transform(features_df), \n",
    "                               index=features_df.index, \n",
    "                               columns=features_df.columns)\n",
    "\n",
    "df = scaled_df.join(df.Decision)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop([\"Decision\"], 1)\n",
    "y = df.Decision\n",
    "\n",
    "# Train, test, split\n",
    "# random_state 随机数种子：该组随机数的编号，在需要重复试验的时候，保证得到一组一样的随机数。\n",
    "# 比如每次都填1，其他参数一样的情况下得到的随机数组是一样的。但填0或不填，每次都会不一样。\n",
    "# train_size: If train_size is None, it will be set to 0.25.\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Function for plotting confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_confusion_matrix(y_true, y_pred, labels=[\"Sell\", \"Buy\", \"Hold\"], \n",
    "                          normalize=False, title=None, cmap=plt.cm.coolwarm):\n",
    "\n",
    "    cm = confusion_matrix(y_true, y_pred)\n",
    "    fig, ax = plt.subplots(figsize=(12,6))\n",
    "    im = ax.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    ax.figure.colorbar(im, ax=ax)\n",
    "    # We want to show all ticks (刻标)...\n",
    "    ax.set(xticks=np.arange(cm.shape[1]),\n",
    "           yticks=np.arange(cm.shape[0]),\n",
    "           # ... and label them with the respective list entries\n",
    "           xticklabels=labels, yticklabels=labels,\n",
    "           title=title,\n",
    "           ylabel='ACTUAL',\n",
    "           xlabel='PREDICTED')\n",
    "    # Rotate the tick labels and set their alignment.\n",
    "    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"right\",\n",
    "             rotation_mode=\"anchor\")\n",
    "    # Loop over data dimensions and create text annotations.\n",
    "    fmt = '.2f' if normalize else 'd'\n",
    "    thresh = cm.max() / 1.5\n",
    "    for i in range(cm.shape[0]):\n",
    "        for j in range(cm.shape[1]):\n",
    "            ax.text(j, i, format(cm[i, j], fmt),\n",
    "                    ha=\"center\", va=\"center\",\n",
    "                    color=\"snow\" if cm[i, j] > thresh else \"orange\",\n",
    "                    size=26)\n",
    "    ax.grid(False)\n",
    "    fig.tight_layout()\n",
    "    return ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling\n",
    "The preferred evaluation metric used will be __Precision__ for each class.  \n",
    "They will be optimized using the __F1 Score-Macro-Average__ to balance the Precision and Recall.  \n",
    "This is done because we want to not only be correct when predicting but also make a decent amount of predictions for each class.  \n",
    "Classes such as 'Buy' and 'Sell' are more important than 'Hold'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fitting and Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None, learning_rate=1.0,\n",
       "                   n_estimators=50, random_state=None)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fitting and training\n",
    "clf = AdaBoostClassifier()\n",
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Print out Evaluation Metrics\n",
    "- micro:  Calculate metrics globally by counting the total true positives, false negatives and false positives.  \n",
    "        通过先计算总体的TP，FN和FP的数量，再计算F1   \n",
    "- macro:  Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.  \n",
    "        分布计算每个类别的F1，然后做平均（各类别F1的权重相同）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        Sell       0.18      1.00      0.31         2\n",
      "         Buy       1.00      0.23      0.38        13\n",
      "        Hold       0.67      1.00      0.80         2\n",
      "\n",
      "    accuracy                           0.41        17\n",
      "   macro avg       0.62      0.74      0.49        17\n",
      "weighted avg       0.86      0.41      0.42        17\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Classifier predictions\n",
    "pred = clf.predict(X_test)\n",
    "\n",
    "#Print out results\n",
    "report = classification_report(y_test, pred, target_names=['Sell', 'Buy', 'Hold'])\n",
    "print(report)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeEAAAGoCAYAAABxHV2qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3dd5xddZn48c8zySSZ9IQECKFJC1WqSFURQVBULCtIE1cWXQt2F9f9YUXXZXVRRN2slaJiAQQFRBeQumiQXgydNAhJCCE9mXl+f9ybOAzTyz23fN6v133l3nO+99wnczJ57rdHZiJJkiqvqegAJElqVCZhSZIKYhKWJKkgJmFJkgpiEpYkqSAmYUmSCmISlroQES0RcWVEPB8RvxzAdU6MiGsHM7YiRMTVEfHuouOQ6olJWDUvIk6IiFkRsTwiFpSTxSGDcOl3AJsBm2TmP/T3Ipl5cWYeOQjxvEhEvCYiMiIu7XB8z/LxG3p5nc9HxEU9lcvMozPzJ/0MV1InTMKqaRHxceBc4CuUEubWwHeAtwzC5bcBZmfm+kG41lB5FjgoIjZpd+zdwOzB+oAo8f8KaQj4i6WaFRETgC8CH8zMSzNzRWauy8wrM/NT5TIjI+LciJhffpwbESPL514TEXMj4hMRsbBci35P+dwXgLOA48o17Pd2rDFGxLblGufw8utTI+KxiHghIh6PiBPbHb+53fsOioi/lJu5/xIRB7U7d0NEfCkibilf59qImNLNj2EtcDlwfPn9w4B3Ahd3+Fl9MyLmRMSyiLgjIg4tHz8K+Nd2f8+728VxdkTcAqwEtisfO618/rsR8at21/9aRPxvRESvb6Akk7Bq2oHAKOCybsp8FjgA2AvYE9gf+Ld25zcHJgDTgfcC50fEpMz8HKXa9SWZOTYzf9BdIBExBvgWcHRmjgMOAu7qpNxk4HflspsA3wB+16EmewLwHmBTYATwye4+G7gAOKX8/PXA/cD8DmX+QulnMBn4KfDLiBiVmdd0+Hvu2e49JwOnA+OAJztc7xPAy8tfMA6l9LN7d7oOrtQnJmHVsk2ART00F58IfDEzF2bms8AXKCWXDdaVz6/LzKuA5cCMfsbTBuweES2ZuSAz7++kzBuBhzPzwsxcn5k/Ax4C3tSuzI8yc3ZmrgJ+QSl5dikzbwUmR8QMSsn4gk7KXJSZi8uf+XVgJD3/PX+cmfeX37Ouw/VWAidR+hJxEfDhzJzbw/UkdWASVi1bDEzZ0BzchS14cS3uyfKxjdfokMRXAmP7GkhmrgCOA94PLIiI30XEzr2IZ0NM09u9frof8VwIfAg4jE5aBspN7g+Wm8CXUqr9d9fMDTCnu5OZ+WfgMSAofVmQ1EcmYdWy24DVwLHdlJlPaYDVBlvz0qba3loBjG73evP2JzPz95l5BDCNUu32f3oRz4aY5vUzpg0uBD4AXFWupW5Ubi7+F0p9xZMycyLwPKXkCdBVE3K3TcsR8UFKNer5wKf7H7rUuEzCqlmZ+TylwVPnR8SxETE6Ipoj4uiI+I9ysZ8B/xYRU8sDnM6i1HzaH3cBr4qIrcuDwj6z4UREbBYRby73Da+h1Kzd2sk1rgJ2Kk+rGh4RxwG7Ar/tZ0wAZObjwKsp9YF3NA5YT2kk9fCIOAsY3+78M8C2fRkBHRE7AV+m1CR9MvDpiOi22VzSS5mEVdMy8xvAxykNtnqWUhPqhyiNGIZSopgF3APcC/y1fKw/n/UH4JLyte7gxYmzidJgpfnAEkoJ8QOdXGMxcEy57GJKNchjMnNRf2LqcO2bM7OzWv7vgaspTVt6klLrQfum5g0LkSyOiL/29Dnl5v+LgK9l5t2Z+TClEdYXbhh5Lql3wsGMkiQVw5qwJEkFMQlLkjRIIuIjEXFfRNwfER/tqbxJWJKkQRARuwP/RGlRoD2BYyJix+7eYxKWJGlw7AL8X2auLK8/8Cfgrd29obtFDmrOmHFTcuLUjlMwVcueX7K86BAkdWPNyqdZt/b5ql8zfN+mMbksO5s12DePsOZ+SjMMNpiZmTPLz+8Dzi4vQ7sKeAOl2RldqqskPHHqNrz/y7cXHYYG0TU/u6XoECR14+6bTi86hF5Zlq2cO3zglbRj1s9enZn7dXYuMx+MiK8Bf6C0VsDdlObod6mukrAkSZ0KiOZBqLD3sLFpebOXHwBExFeAbtdUNwlLkjRIImLTzFwYEVsDb6O021uXTMKSpLoXETQNr0jX9a/LfcLrKO11/lx3hU3CkqT6FxDNQz8hKDMP7Ut5k7Akqf4FlaoJ94nzhCVJKog1YUlS/Rus0dGDzCQsSap7FRyY1ScmYUlS/avSmrB9wpIkFcSasCSp/lXp6GiTsCSp7gUQw6ovCdscLUlSQawJS5LqX0BTFdaETcKSpAYQRJNJWJKkyguIYdXXA1t9EUmS1CCsCUuS6l5gn7AkScUI7BOWJKkYUZU1YfuEJUkqiDVhSVLdi6jOFbNMwpKkhhBN1df4W30RSZLUIKwJS5Lqn6OjJUkqSnWOjjYJS5LqXlRpTdg+YUmSCmJNWJLUEKpxdLRJWJJU/6q0OdokLElqANU5MKv66uaSJDUIa8KSpLpXraOjTcKSpIbgwCxJkopQoZpwRHwMOA1I4F7gPZm5uqvy1fe1QJKkGhQR04EzgP0yc3dgGHB8d++xJixJagBRqT7h4UBLRKwDRgPzeyosSVLdG6QkPCUiZrV7PTMzZwJk5ryI+E/gKWAVcG1mXtvdxUzCkiT13qLM3K+zExExCXgL8DJgKfDLiDgpMy/q6mL2CUuS6l5pilLTgB89eB3weGY+m5nrgEuBg7p7gzVhSVJDqMCKWU8BB0TEaErN0YcDs7p7g0lYklT/YugHZmXm7RHxK+CvwHrgTmBmd+8xCUuSNEgy83PA53pb3iQsSWoIrpglSVIBXDta/TKcVWzP79mea9mCWUziMZpZwWomspA9eJC3cjensp6WokNVPxw6/SaO3fEKdpr0CGOal7No1RRuX7A/P3/oncxbPr3o8NRH3s/qZhJWn32c6YzkhZccH8MiXsb1vIzr2Z/vcAmXsoQdC4hQ/ZOcuf85vGn7q150dPrYBbxtx99w1LbXctYtn+O2BQcUFJ/6xvup/qm+BnK9yEheYD0juY/j+TUXcx5/4xwWMpNZzOJ9JMEUHuIE3kAzy4sOV710yq4Xb/wP+3+fPIxTrv4Bb7z0cv7lxrOZv3xzRjev4osHf4Gtxs0pOFL1hvezFkQl5gn3mUm4yv2Ff+ZbPMplXMgDvJOlbMdqJvEMe3I13+Y6zgZgEk+wH98rOFr1xqRRSzh519ICOrfMO4Czbj2LR5duz9I1E7l53sGccd1/sXLdKEY3r+L0l3+/4GjVE+9njSj3CQ/0MdhMwlXuGr7FCjbr8vxtfIyVbALADlxTqbA0AEdvey2jm0s7m8285zTgxb/YC1ZM48pHjwHg1VvexKRRSyodovrA+6mBMAnXuGQ4S9gBgHEsKDga9cbB028F4KllW/LI0h06LXP9nFcDMKypjQOn3V6x2NR33s9aYXO0hsgYFgKwhnEFR6Le2GnSwwA8sHiXLss8tGQG69tKv54zJs+uSFzqH+9nDYkY+GOQmYRr3ObcySQeB2Auryw4GvVkSsuzjG5eBcD85Vt0WW5d2wgWryp1M2wz/qmKxKa+837WjrBPWEPhcM4EIAnu5LSCo1FPJo58fuPzpWsmdFv2uTWTABg/YtmQxqT+835qoCqahCPisxFxf0TcExF3RUSXVbeI+HFEvKP8/IaI6HT/xkZ2IF9nO64DYBbvYyF7FByRejJq+OqNz9e2jui27Jry+Zbhq4Y0JvWf97O2VGOfcMUW64iIA4FjgH0yc01ETAG6/1erLm3HtbyWzwLwDLvzR/6j4IjUG0FufJ5037QVnTxTdfF+1pAK7KLUH5VcMWsasCgz1wBk5iKAiNgX+AYwFlgEnJqZDvPtxubcwTs4niZaeZ6t+DlXuGxljVi1/u/3aeSwNd2WHTFsbfk9o4Y0JvWf97O2VOMGDpWM6Fpgq4iYHRHfiYhXR0QzcB7wjszcF/ghlFef6KWIOD0iZkXErBUvLBqCsKvLZGZzAm9iJC+wgqlczFUsY6uiw1Ivte83bN+f2JmJI5cCsGzt+CGNSf3n/dRAVawmnJnLy7XeQ4HDgEuALwO7A3+I0tDvYdC3ya6ZOZPypsnTt9s3eyhe08YzhxM5mjE8y2rG81N+y2J2Ljos9cGiVVNZua6F0c2rmDa263/qzU1rmdJS+lL55LKtKxWe+sj7WVsavTmazGwFbgBuiIh7gQ8C92fmgZWMoxaN5llO5Ggm8hTraOESLudp9ik6LPXD7Od2ZK9N72G3TR7sssyMybMZ3tQGwN+W7FSp0NQP3s/aUK1bGVasOToiZkRE+21+9gIeBKaWB20REc0RsVulYqoVI1jGCbyRKfyNVpr5FZfwFIcWHZb66ZZ5BwGw9fg5bDfhsU7LHLbVnwBobWvitgXO/65m3s9aEdDUNPDHIKtkn/BY4CcR8UBE3APsCpwFvAP4WkTcDdwFHFTBmKreMFZzPMcyjTtpo4nL+TGPcHTRYWkArn7iyI2Dc97XyYL+m49ZwJu3vxKAP809lOdWT65ofOob76cGopJ9wnfQeYJdBLyqk/Kntnv+miELrIoFrbydE9iGmwD4I1/j4W62LEyaWM/oSoaofnhu9WQuuP8k3rfn9zlky1v5wkFf4Cf3n8yS1ZPZdZMH+Oi+5zG6eTUr17WUNwRQNfN+1o4YgmUnB6qifcLqm/HMYQZXbnx9JJ/iSD7VZfmlbMN5PFKJ0DRAFzxwIluMnc+btr+K121zPa/b5voXnV+5roWzbvkcc15w5Hst8H7WgKjOKUomYakQwb//+dPcOv9A3rLDFcyY9DCjm1ewaNUU/rzgFfzsoeOYt3x60UGq17yf6h+TcBV7nm35EuuKDkND6Ma5h3LjXAfZ1QvvZzVzxSxJkooRDMno5oEyCUuSGkI11oSr72uBJEkNwpqwJKnuBUFE9dU7TcKSpPoXQBU2R5uEJUkNoRrnCVdfRJIkNQiTsCSpIURTDPjR42eUNiu6q91jWUR8tKvyNkdLkupfaS/DIf+YzPwbpV0CiYhhwDzgsq7KWxOWJGloHA48mplPdlXAmrAkqSEM0mIdUyJiVrvXMzNzZhdljwd+1t3FTMKSpMYwOKOjF2Xmfj0ViogRwJuBz3RXziQsSap7EVHp/YSPBv6amc90V8g+YUmSBt+76KEpGqwJS5IaRYUW64iI0cARwPt6KmsSliQ1hErtopSZK4FNelPWJCxJqn8VmifcV9UXkSRJDcKasCSpMbiLkiRJxXA/YUmSilCl+wlX39cCSZIahDVhSVIDCKJC84T7wiQsSWoMlV22sleq72uBJEkNwpqwJKn+BRVbtrIvTMKSpAYQVdkcbRKWJDWEahyYVX0RSZLUIKwJS5LqX1CVGziYhCVJDSCqcsUsk7Akqe4F1bl2dPVFJElSg7AmLEmqf1W6gYNJWJLUAKIqB2ZVX0SSJDUIa8KSpMbgilmSJBWkClfMMglLkupf2CcsSZLasSYsSWoMTlGSJKkgVdgcbRKWJDWGKhwdXX1fCyRJahDWhCVJ9S/CKUqSJBXG5mhJkgoSTQN/9OZjIiZGxK8i4qGIeDAiDuyqrDVhSZIG1zeBazLzHRExAhjdVUGTsCSp/lWoTzgixgOvAk4FyMy1wNquytdVEh79+P3sfcpuRYehQbTzTQ8WHYIG2RW/e7roEDSIHpo1qugQem9w+oSnRMSsdq9nZubMdq+3A54FfhQRewJ3AB/JzBWdXcw+YUmSem9RZu7X7jGzw/nhwD7AdzNzb2AFcGZXFzMJS5IaQ2UGZs0F5mbm7eXXv6KUlDtlEpYkNYAo76Q0wEcPMvNpYE5EzCgfOhx4oKvyddUnLElSp4JKLtbxYeDi8sjox4D3dFXQJCxJ0iDKzLuA/XpT1iQsSap7CWQVrphlEpYkNYBwK0NJkgpThUm4+iKSJKlBWBOWJDUE+4QlSSpCVGefcPVFJElSg7AmLElqDDZHS5JUkMqtmNVrJmFJUgOIqhyYVX1fCyRJahDWhCVJ9S+oytHRJmFJUkNIk7AkSUXo3X7AlVZ9XwskSWoQ1oQlSQ3B5mhJkopShc3RJmFJUv1z7WhJktSeNWFJUt1L3MpQkqTi2BwtSZI2sCYsSWoIic3RkiQVIJwnLElSYaowCVdfRJIkNQhrwpKk+hdOUZIkqRBpn7AkSQWqwppw9X0tkCSpQVgTliQ1hEo1R0fEE8ALQCuwPjP366qsSViS1ACi0ot1HJaZi3oqZBKWJDWEahyYVX0RSZJUvaZExKx2j9M7KZPAtRFxRxfnN7ImLEmqf8FgjY5e1F0fb9nBmTk/IjYF/hARD2XmjZ0V7FdNOCI+2p/3SZJUjCBpGvCjNzJzfvnPhcBlwP5dle1vc/TH+/k+SZLqVkSMiYhxG54DRwL3dVW+v83R1TfjWZKkLiQVW7ZyM+CyKH3WcOCnmXlNV4X7m4Szn++TJKkQlRgdnZmPAXv2tnyXSTgiXqDzZBvA6L6HJklScSo8T7hXukzCmTmukoGoey3bTGe7j5zC1NceSMtW02hbu44Vj81hwWXX8vj5F9O2anXRIaoXWtbPYYuVVzB5zV+YsPY+RrU+w4jWRbTFCFYO35aFLa/h0XH/zAsjdi46VPVassWIR9iu5U62b7mL7VruYquRD9LctJa1bSN570OPFR2gqlifmqPLnczHAidk5huHJiR1NO0dR7HXzC8zrGXUxmPDRrcwcZ/dmLjPbmx1ylu5/ZjTWfXU/AKjVG9ssfIK9ll8xkuOD8u1TFh3HxPW3cf2y/6buzb5Oo+O/2ABEaqvpjTP5Ws7vKboMNSjGt1FKSJGAG8ATgCOAn4NfG+I41LZpAP2Yu8ffpWm5mZWzXuaBz/7DRbfcDs0NTH1dQezy5c/xtgdt2X/y77DTQe9k7Y1a4sOWd1ojVHMb3kDC1tey9IRe7Fq2BasHTaFka3PsMma29h56X8wbv0j7LP4DFYM35anR/tdt5YsWbc5j67am3HDlrDzmNuLDkcd1NR+whFxBPAu4PXA9cCFwP6Z+Z4KxSZgl69+kqbmZtrWruP/jj6NFQ8/sfHc3Asv54X7ZnPIzT9n3K47sO3738Vj3/xJccGqR0+Mey9PjHvvS46vHbYJL4zYlXmj387r5+1BS+t8Zjz/dZNwDVjeOon/euqHPLpqb55v3RSAt079ukm4yiTV2SfcXd3898D2wCGZeVJmXgm0VSYsAYzacnMmH7AXAHMv+s2LEvAGz9/5AAsu/wMA27zv+EqGpyGwbthE5o55GwCT1txRcDTqjdVtY/nr8tdvTMBSX3SXhPcF/g/4Y0T8ISLeCwyrTFgCGL/HjI3Pn7v97i7LLf3zPQCMedlWjNt9pyGPS0OrjebSnzGqh5KSei1KfcIDfQy2Lq+YmXdm5r9k5vbA54G9gRERcXVPC1JrcDRPGLvx+eoFC7ss1/7chL13HdKYNLSa2lazxcorAVgyct+Co5HqS5a3MxzIY7D1Kq1n5i2Z+SFgOnAucOCgR6KXWLds+cbno6Z13dTV/tzYGS8b0pg0BLKNUesXMG3FFRy24DWMW/8IrYzg/kmfLzoySUOsu4FZ+3Q4lJR2j/g9pf5iDbHlD/59fuHEV+zBnAsu67TcxFfssfH5iE0mDXlcGhyHPn00m6+69iXHlzXP4I5NvstzI7tc811SP9TaFKWvd3JscnnK0vGZ2XUnZSciohW4l9KKW63AhzLz1r5co9GsfHwOy+6dzfg9dmLLk4/lsXN/zIpHn3pRmfF77sy0Y4/Y+Hr4WBczq2Wrm6by8PgzeM6maGnQVePo6O5WzDqss+MRsR9wHvCqPn7Wqszcq3yN1wNfBV7dx2s0nIfOOpf9L/sOw0aO4ICrf8CDn/0Gi/70Z6IpmHr4Qexy9icggtY1axk2cgTZ5gD2WnHLppcRrCdoY0TrYqasvpldnv8a+y7+IDssO5+bN7uClc12L0iDIat0sY4+R5SZs4CxPRbs3njgOYCIeE1E/HbDiYj4dkScGhGHR8Rl7Y4fERGXDvBza87Ca27k/k/+O9naSstW09jngnM48sk/ccTjN7DX97/CiKmTeODMczYuW7n++eU9XFHVoq1pFK1NY1nfNJ6VzS/jqXEn84ct/sLika9kwroHOOSZYyH9UiXVsz4n4YjYjP7totQSEXdFxEPA94Ev9VD+OmCXiJhafv0e4Ef9+Nya9/j5F3HTwccx9+IrWPnUfFrXrGXNs0t45nc3cNvr/5E5P76U5onjAVg19+mCo9VAtDW1cO+krwAwYd19bLr6uoIjkupHNY6O7m5g1nm8NNlOBg4CPtKPz2rfHH0gcEFE7N5V4czMiLgQOCkifkRpRPYpncR5OnA6wNR+78xY/Zbd/RB3nfavnZ7b5FWv2Pj8+b92uXe0asTika/c+HzimrtY2PK6AqOR6kdNLVsJzOrwOoHFwMczs+tJq72QmbdFxBRgKrCeF9fI269Q8CPgSmA18MvMXN/JtWYCMwF2jFENuc/xZseUuu/Xv7CCJbfdWXA0Gqgm2v0zr8L/NKRalVl9v0/dJeHDMvPUofjQiNiZ0upbi4EngV0jYiSlBHw4cDNAZs6PiPnAvwFHdHG5htY8eQJbnvgWAOb/8mpaV6wqOCIN1JTVN258vmK4A7OketZdEn75IH9WS0TcVX4ewLszsxWYExG/AO4BHgY6VuUuBqZm5gODHE/Ni2HD2Gvm2YyYPIF1y5Yz+yvfLTok9WDc2oe63Su4ufU5Xr7kMwCsi3E8Y1O0NEiC7PswqCHXXRIeHRF7Q+c90Zn51758UGZ2ue50Zn4a+HQXpw8B/qcvn1VPWraexr4/O5c5F1zG0r/cy6qn5tM0aiSTXrknO3zyNCbstQvZ2sq9H/4Cq+c9U3S46sGR817OgtHHMG/0sTw3cl9WD9uMpImW1vlsuup6Zjz/dUa3zgHg3slfYX3T+IIjVm9sMWI2LcNe2Ph68vAFQOm//e1bXrwRx5Ord2d9jqxofKreXZS6S8LTKS3Y0VnUCbx2SCJqJyLuAFYAnxjqz6pewcR9dmPiPrt1enbdsuXc95EvMf8XV1c4LvVHE61MX/kbpq/8TZdlWmMU9046m0fHf6CCkWkgTp32r+wy5raXHG9uWsvnX/bmFx372MP/x6J1W1UqNLVTa0n4kcwc8kTbncxs+GWD1ixczANnnsOUww5g7C7bM3LqZNrWrGXVUwt45uo/8eTMn7N6/oDGyamCrp92A5uuup4pq29izPonGdn6DE25lnVNE3hhxC4sHPUaHh/3HlYN37roUCVVQP3O6akTbavX8Ng3f8Jj3/xJ0aFoECwadSiLRh1adBgaZF958ldFh6BeqMaacHe91F+NiJfsixcRu7VbQEOSpBow8IU6Kr2V4dsozePtaEvgm4MeiSRJDaa7JLxHZv6p48HyVoaDPX1JkqQhlRkDfgy27vqER3RzrnmwA5EkaahU6xSl7mrCsyPiDR0PRsTRwGOdlJckqWpVY59wdzXhjwG/jYh3Ahtmm+9HaSOFYwY9EkmSGkyXNeHMnA3sAfwJ2BbYBrgB+Ef6t4uSJEmFqbWaMJm5BvhRefnKdwGfAx4Hfj3okUiSNGSGZmDVQHW3n/BOwPGUku9i4BIgMvOwCsUmSdKgSKCtQgOzImIYpe2A52Vmt9233dWEHwJuAt6UmY+UL/yxQYtSkqT69BHgQaDHHVi6Gx39duBp4PqI+J+IOJwudlSSJKnaVaJPOCK2BN4IfL83MXU3MOuyzDwO2JnSgKyPAZtFxHcj4sjeXFySpKqQg7ZYx5SImNXucXqHTzqX0ta8bb0Jq8cNHDJzBXAxcHFETAb+ATgTuLYvf39JkurAoszcr7MTEXEMsDAz74iI1/TmYt01R79EZi7JzP8ueotDSZL6qgLN0QcDb46IJ4CfA6+NiIu6e0OfkrAkSbVp4E3RPU1xyszPZOaWmbktpdlF12XmSd29x/2EJUl1r1rXjjYJS5I0yDLzBkqDmrtlEpYkNYSaWjFLkqR60qs5QxVmEpYkNYRqrAk7OlqSpIJYE5Yk1b2h2opwoEzCkqSGUI3N0SZhSVJDqMaasH3CkiQVxJqwJKn+JbRl0UG8lElYklT3qnXZSpujJUkqiDVhSVJDcHS0JEkFSfuEJUkqQtBmn7AkSdrAmrAkqe4l9glLklQY+4QlSSqI84QlSdJG1oQlSfXPZSslSSpGtQ7MsjlakqSCWBOWJDUER0dLklSQalwxyyQsSWoI1VgTtk9YkqSCWBOWJNW9JKpydLRJWJJU/5wnLElScewTliRJG1kTliQ1hEps4BARo4AbgZGUcuyvMvNzXZU3CUuS6l5SsT7hNcBrM3N5RDQDN0fE1Zn5f50VNglLkhpCJfqEMzOB5eWXzeVHl59cV0l4wYRt+OqhM4sOQ4PpKzcVHYEG2VHvOrjoEDSIrmupqzTSG1MiYla71zMz80WJJyKGAXcAOwDnZ+btXV2s4X56kqTGNEg14UWZuV/3n5OtwF4RMRG4LCJ2z8z7OitrEpYk1b1MaKvwYh2ZuTQibgCOAjpNwk5RkiRpkETE1HINmIhoAV4HPNRVeWvCkqSGUKHFOqYBPyn3CzcBv8jM33ZV2CQsSWoIFRodfQ+wd2/Lm4QlSQ2hGteOtk9YkqSCWBOWJNW9BLcylCSpEFmduyiZhCVJDcE+YUmStJE1YUlS3Sv1CRcdxUuZhCVJDaEak7DN0ZIkFcSasCSpIVTjwCyTsCSp/jlFSZKkYiTQ1lZ0FC9ln7AkSQWxJixJagg2R0uSVBCTsCRJBcisztHR9glLklQQa8KSpIaQVdgebRKWJDWEKszBJmFJUmNwnrAkSdrImrAkqe6ly1ZKklQcpyhJkqSNrAlLkhqCzdGSJBUkq7A92iQsSap7LlspSZJexJqwJKkh2CcsSVJB2qqwPdrmaElS3Uv+vmDHQB49iYitIuL6iHgwIu6PiI90V96asCRJg2c98InM/GtEjAPuiIg/ZOYDnX+px+MAABG1SURBVBU2CUuS6l+Flq3MzAXAgvLzFyLiQWA6YBKWJDWqpG1wsvCUiJjV7vXMzJzZWcGI2BbYG7i9q4uZhCVJ6r1FmblfT4UiYizwa+Cjmbmsq3ImYUlSQ8gK7SccEc2UEvDFmXlpd2VNwpKkulcaHT30ncIREcAPgAcz8xs9lTcJS5LqX0JbZWrCBwMnA/dGxF3lY/+amVd1VtgkLEnSIMnMm4HobXmTsCSpIVSiObqvTMI15NDpN3Hsjlew06RHGNO8nEWrpnD7gv35+UPvZN7y6UWHp37wnta+4axie37P9lzLFsxiEo/RzApWM5GF7MGDvJW7OZX1tBQdakNLqnMXJZNwTUjO3P8c3rT9i7sUpo9dwNt2/A1HbXstZ93yOW5bcEBB8anvvKf14uNMZyQvvOT4GBbxMq7nZVzP/nyHS7iUJexYQIQCSot1VGEWdu3oGnDKrhdv/M/6f588jFOu/gFvvPRy/uXGs5m/fHNGN6/iiwd/ga3GzSk4UvWW97R+jOQF1jOS+zieX3Mx5/E3zmEhM5nFLN5HEkzhIU7gDTSzvOhwVWVMwlVu0qglnLzrRQDcMu8Azrr1LB5duj1L10zk5nkHc8Z1/8XKdaMY3byK01/+/YKjVW94T+vLX/hnvsWjXMaFPMA7Wcp2rGYSz7AnV/NtruNsACbxBPvxvYKjbWyV2MChr0zCVe7oba9ldPNqAGbecxodB90tWDGNKx89BoBXb3kTk0YtqXSI6iPvaX25hm+xgs26PH8bH2MlmwCwA9dUKix1oq0tB/wYbCbhKnfw9FsBeGrZljyydIdOy1w/59UADGtq48BpXS5RqirhPW0syXCWULrP40rr+qsAmTkoj8FmEq5yO016GIAHFu/SZZmHlsxgfVvpVs6YPLsican/vKeNZwwLAVjDuIIjUbUxCVexKS3PMrp5FQDzl2/RZbl1bSNYvKrU3LXN+KcqEpv6x3vaeDbnTibxOABzeWXB0TS2bBv4Y7CZhKvYxJHPb3y+dM2Ebss+t2YSAONHdLlZh6qA97TxHM6ZACTBnZxWcDSNrS1zwI/BNmRJOCKWd3h9akR8u4f3fD4iPtnJ8W0j4r7BjrHajRq+euPzta0jui27pny+ZfiqIY1JA+M9bSwH8nW24zoAZvE+FrJHwRGp2rhYRxUL/v6tK3tYijQ6eabq4z1tHNtxLa/lswA8w+78kf8oOCK5bGVZRGwD/BCYCjwLvCczn+pQZt9ymZXAzRUPsgqsWv/3Ze5GDlvTbdkRw9aW3zNqSGPSwHhPG8Pm3ME7OJ4mWnmerfg5V7hsZcEyGZIpRgM1lH3CLRFx14YH8MV2574NXJCZLwcuBr7Vyft/BJyRmQd29yERcXpEzIqIWevWPt9d0ZrTvs+wfV9iZyaOXArAsrXjhzQmDYz3tP5NZjYn8CZG8gIrmMrFXMUytio6LNF4i3Wsysy9NjyAs9qdOxD4afn5hcAh7d8YEROAiZn5p3ZlOpWZMzNzv8zcr3lE9wNdas2iVVNZua707Xna2K7nFzY3rWVKyyIAnly2dUViU/94T+vbeOZwIkczhmdZzXh+ym9ZzM5Fh6UqVi2jozt+v4hOjjWk2c+VFnzfbZMHuywzY/JshjeVxs7/bclOFYlL/ec9rU+jeZYTOZqJPMU6WriEy3mafYoOS+1kWw74MdiKSsK3AseXn59Ihz7fzFwKPB8Rh7Qr05BumXcQAFuPn8N2Ex7rtMxhW5UaDFrbmrhtgfMQq533tP6MYBkn8Eam8DdaaeZXXMJTHFp0WGonB2F6Uk1NUerBGcB7IuIe4GTgI52UeQ9wfkTcBjTsHI2rnzhy48Cc93WymP/mYxbw5u2vBOBPcw/ludWTKxqf+s57Wl+GsZrjOZZp3EkbTVzOj3mEo4sOS51oqJpwZo7t8PrHmfmh8vMnMvO1mfnyzDx8w8jozPx8Zv5n+fkdmblnZh5YPr77UMVazZ5bPZkL7j8JgEO2vJUvHPQFtpvwGBNHLuWgLW7lW6/9OKObV7NyXUt5MwBVO+9p/QhaeTsnsA03AfBHvsbD5S0LO3sMZ2XBEavaOE+4BlzwwIlsMXY+b9r+Kl63zfW8bpvrX3R+5boWzrrlc8x5wRGYtcJ7Wh/GM4cZXLnx9ZF8iiP5VJfll7IN5/FIJUJTJ4aiJjtQJuGaEPz7nz/NrfMP5C07XMGMSQ8zunkFi1ZN4c8LXsHPHjqOecunFx2k+sR7KlVUQhXmYJNwLblx7qHcONfBHvXEe1rbnmdbvsS6osNQDTMJS5LqXmJztCRJBUnXjpYkqRANuHa0JEnqhjVhSVJDsDlakqQCODBLkqSiZHUmYfuEJUkqiElYktQAKrOLUkT8MCIWRsR9vYnKJCxJaggV2kXpx8BRvY3JPmFJUt1LKjM6OjNvjIhte1veJCxJUu9NiYhZ7V7PzMyZ/b2YSViSVP8Gb8WsRZm532BcCEzCkqQG4RQlSZK0kUlYktQASrsoDfTRk4j4GXAbMCMi5kbEe7srb3O0JKnuZUK2tVXgc/JdfSlvEpYkNQS3MpQkSRtZE5YkNQS3MpQkqQjZ62UnK8okLEmqe9W6n7B9wpIkFcSasCSpIbTl0E9R6iuTsCSp/qXN0ZIkqR1rwpKkupc4OlqSpMI4T1iSpCIktFVg7ei+sk9YkqSCWBOWJDUE+4QlSSpAkqTzhCVJKoDzhCVJUnvWhCVJDaEaa8ImYUlSA0jXjpYkqQhpn7AkSWrPmrAkqSFkFa6YZRKWJNU/m6MlSVJ71oQlSQ3AFbMkSSpEAm1V2BxtEpYk1b+szoFZ9glLklQQa8KSpAaQjo6WJKkomW0DfvQkIo6KiL9FxCMRcWZP5a0JS5LqXwXmCUfEMOB84AhgLvCXiLgiMx/o6j3WhCVJGhz7A49k5mOZuRb4OfCW7t5gTViSVPeSrMTo6OnAnHav5wKv7O4NkVl9HdX9FRHPAk8WHUcFTAEWFR2EBpX3tP40yj3dJjOnFh1ETyLiGkr3ZKBGAavbvZ6ZmTPLn/EPwOsz87Ty65OB/TPzw11drK5qwrXwD2EwRMSszNyv6Dg0eLyn9cd7Wl0y86gKfMxcYKt2r7cE5nf3BvuEJUkaHH8BdoyIl0XECOB44Iru3lBXNWFJkoqSmesj4kPA74FhwA8z8/7u3mMSrk0ziw5Ag857Wn+8pw0oM68Crupt+boamCVJUi2xT1iSpIKYhCVJKohJuMZERBQdg6SuRcTEomNQ7TAJ15CIeAVwSkS0FB2LBl9EOFCyxkXEdOCWiHht0bGoNpiEa8sY4EPA2yNiVNHBaPBExE7AdyNiZNGxqH8iIjJzHnAOcE5EHFB0TKp+JuEaEBF7RMTJmXkD8AngNOCdJuLa1657YQTQRmluoWpMOQFvmGryBKXlc2dGxEHFRaVaYBKuDXsAb42IEzLzRuDzwD9iIq4H48t//g3YDPhcgbGonzYk4PJCDWdT2j3nOuB7EfGqImNTdTMJV7ENtaTM/CnwS+CNEXFSuUb8eUqJ+B32EdemiNgSuCAi3puZ6yh1NYyJiG0KDk29FBEzIuLodod2BP5fZv4C+DjwPeC/IuKQQgJU1XMgSJXq0LxFZv4sIpYBJ0cEmXlRRJwFfAtYB1xSVKzqu4jYmtK2Z98APhkRL6dUE24Bdgae7PhvQNUlIpqBtwPTy7fqGkr37yTgusxsi4jrgHcB/x4RR2TmqgJDVhVyxawqFxH/BGxNaeus84FDKC0KflVm/jQiDgbmZmYjbOFY8yKiCZgA/DulfUe/DgQwkVJ//+uAF4B3ZObTRcWp3omIzYFTgGmUvgg/SGnJwtsz8+MRcRywO/DtzHymuEhVrUzCVSYiRmfmyvLzM4A3A18EzgV+nZlnl/esPAW4sNzspSrXsVZbnsLyD8BjwGWZ+Uj5+G7A+4AfZObdhQSrbnVyL6dS6hrailJf8MPAryltYbcHpS9U3S7ir8Zln3AViYg3AF+JiK0iYhilX+rXA/sBT1Oa9jAiM38J/DdwS3HRqi8yMyNiz4g4r/z6OuBnlPoQj4uI7crH76e0B+mRhQWrLrVPwBHxpog4CpiRmV+jNCL6OGDrzDwEOBU4xASs7piEq0REHAN8FbghM+dQmq6yJXADpSbot2TmWuAfI+LNmfnb8pxEVamI2D4i3hYRx5YPrQMmR8R/lf8zvxH4HfB+4G0RMTEixlBqmu71LiyqvIj4AKUWqkOA/4mIz2bmOZSmJ70/Ig7PzJWZubjIOFX9TMJVoNyv9AngtMy8PCJGlb9t/5hSX9NFmbkuIk4FPgL4zbrKlRff+A1wMPDpiPjHzHyA0vSVCZS6FwDuBu4ErsnMpZm5Ajja2lN1iYgdImJCuUVjU0pdCSdk5r8BB1H6cnwq8H3gPuDe4qJVLXF0dHVYQ6mWtLo87/fMiHg1pQE6SyhN+j8a2At4e2Y+Wlyo6klE7ApcDHwmM6+MiJOA8RGxW2beHxH/AZwdEbdRqvV+NDPva9fUubbA8NVBREwCPgisjYivZubCiFhM+T5l5nMR8THg4Mz8cUR8KzNbi4xZtcOBWVWgPB/445T6AXcD/gjcDDwAHAvMBi4DmjLz2aLiVO+U54TemJlN5df3APOALYA7M/PU8vE3AvMy866iYlXXNnwpKv9+HkWpxtsKfAH4CnAEcEBmro+IDwMHUBow2ebUMvWWNeEqUP5F/2/gVkqDsX6TmWsAIuJ04B77lmpHZt4cEW+MiMcojX7+VWZ+MSJGAPdGxL9l5pcz83cFh6ruDQPWU6qsXB0R44FPAysy8zMRMQ64sfwl65XAidaA1VfWhKtYeSrSmcA7bYKuPRFxOPB7YERmtpWPvReYmJlfLzQ4dSsipgCzgP3Lzc9bUFq17m5gOfBcZn41Ivah1Mf/RGY+XlzEqlUOzKpCETEtIj5KaWnKd5uAa1Nm/i+led6zoTS4B/gUDtqpepm5CPgwcF1E7A5cCPw0Mz9AaeT6phHxNeCRzLzeBKz+sjm6Oi2lNOH/LRsWcVBtysyrIqItIlYCj1MahHVt0XGpZ+VBdeuAe4B/zczzy6duAkYCh5b/lPrN5mipAspN0+Mz87KiY1HfRMQRwHnAKzPz+XbHN65uJ/WXSViqIDdlqE3lKYLnAgdm5pKi41H9sDlaqiATcG0qj44eAfwxIvYrHfJeauCsCUtSL0XE2MxcXnQcqh8mYUmSCuIUJUmSCmISliSpICZhSZIKYhKWJKkgTlGSuhARrZSWmBwOPEhpCdGVHY4/DpycmUsjYttyub+1u8w3MvOCiHiC0taUUNoY4FLgS5m5pvy+32bm7uXP3R/4T2AzICntqHUn8E/l9+9a/oxW4BrgIeAcSjs1bXACsLIcz0PAqPLnn5+ZPxngj0bSIHF0tNSFiFiemWPLzy8G7sjMb3Q4/hNgdmae3TGZdrjWE8B+mbkoIsYCM4F1mfnu9u+LiM2APwPHZ+Zt5W303g7clJnPdLxW+fWp5dcf6vCZL4onIrajlPy/mZk/GqQfk6QBsDla6p2bgB06OX4bML0vFyrPM30/cGxETO5w+oPATzLztnLZzMxfbUjAA5GZj1Hat/qMgV5L0uAwCUs9iIjhwNF02P0oIoYBhwNXtDu8fUTc1e5xaGfXzMxllJqyd+xwanfgjn6EeVyHz23potxfgZ37cX1JQ8A+YalrLRFxV/n5TcAPOhzfllLC/EO79zyamXv18voxKFGWXNJJc/RQf6akAbImLHVtVWbuVX58ODPXtj8ObAOMoNSE3CcRMY5SEp/d4dT9wL4DiLkne1MarCWpCpiEpX4qb2t3BvDJiGju7fvKA7O+A1yemc91OP1t4N0R8cp25U+KiM0HGm95oNZ/UtqWT1IVsDlaGoDMvDMi7gaOp9RkvX27JmyAH2bmt8rPry+Pdm4CLgO+1Mn1nomI44H/jIhNgTbgRkqjmrtzXEQc0u71B4D55Xju5O9TlM5zZLRUPZyiJElSQWyOliSpICZhSZIKYhKWJKkgJmFJkgpiEpYkqSAmYUmSCmISliSpIP8fISLGbSVLMIgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_confusion_matrix(y_test, pred, title=\"Confusion Matrix\")\n",
    "# Plot non-normalized confusion matrix\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tuning Model Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameters to Tune\n",
    "params = {'n_estimators': [50, 100, 200, 500],\n",
    "          'learning_rate': [1, .1, .01]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 12 candidates, totalling 36 fits\n",
      "[CV] learning_rate=1, n_estimators=50 ................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "C:\\Users\\72445\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.4s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.5s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  learning_rate=1, n_estimators=50, score=(train=0.874, test=0.286), total=   0.5s\n",
      "[CV] learning_rate=1, n_estimators=50 ................................\n",
      "[CV]  learning_rate=1, n_estimators=50, score=(train=0.875, test=0.412), total=   0.1s\n",
      "[CV] learning_rate=1, n_estimators=50 ................................\n",
      "[CV]  learning_rate=1, n_estimators=50, score=(train=0.827, test=0.468), total=   0.4s\n",
      "[CV] learning_rate=1, n_estimators=100 ...............................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:    1.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  learning_rate=1, n_estimators=100, score=(train=0.910, test=0.222), total=   0.2s\n",
      "[CV] learning_rate=1, n_estimators=100 ...............................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   4 out of   4 | elapsed:    1.2s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  learning_rate=1, n_estimators=100, score=(train=0.841, test=0.401), total=   0.2s\n",
      "[CV] learning_rate=1, n_estimators=100 ...............................\n",
      "[CV]  learning_rate=1, n_estimators=100, score=(train=0.879, test=0.363), total=   0.4s\n",
      "[CV] learning_rate=1, n_estimators=200 ...............................\n",
      "[CV]  learning_rate=1, n_estimators=200, score=(train=0.946, test=0.360), total=   0.3s\n",
      "[CV] learning_rate=1, n_estimators=200 ...............................\n",
      "[CV]  learning_rate=1, n_estimators=200, score=(train=0.928, test=0.350), total=   0.5s\n",
      "[CV] learning_rate=1, n_estimators=200 ...............................\n",
      "[CV]  learning_rate=1, n_estimators=200, score=(train=0.879, test=0.420), total=   0.4s\n",
      "[CV] learning_rate=1, n_estimators=500 ...............................\n",
      "[CV]  learning_rate=1, n_estimators=500, score=(train=0.946, test=0.340), total=   1.0s\n",
      "[CV] learning_rate=1, n_estimators=500 ...............................\n",
      "[CV]  learning_rate=1, n_estimators=500, score=(train=0.946, test=0.431), total=   0.8s\n",
      "[CV] learning_rate=1, n_estimators=500 ...............................\n",
      "[CV]  learning_rate=1, n_estimators=500, score=(train=0.877, test=0.511), total=   0.7s\n",
      "[CV] learning_rate=0.1, n_estimators=50 ..............................\n",
      "[CV]  learning_rate=0.1, n_estimators=50, score=(train=0.774, test=0.217), total=   0.1s\n",
      "[CV] learning_rate=0.1, n_estimators=50 ..............................\n",
      "[CV]  learning_rate=0.1, n_estimators=50, score=(train=0.877, test=0.230), total=   0.1s\n",
      "[CV] learning_rate=0.1, n_estimators=50 ..............................\n",
      "[CV]  learning_rate=0.1, n_estimators=50, score=(train=0.807, test=0.481), total=   0.1s\n",
      "[CV] learning_rate=0.1, n_estimators=100 .............................\n",
      "[CV]  learning_rate=0.1, n_estimators=100, score=(train=0.798, test=0.287), total=   0.1s\n",
      "[CV] learning_rate=0.1, n_estimators=100 .............................\n",
      "[CV]  learning_rate=0.1, n_estimators=100, score=(train=0.877, test=0.263), total=   0.2s\n",
      "[CV] learning_rate=0.1, n_estimators=100 .............................\n",
      "[CV]  learning_rate=0.1, n_estimators=100, score=(train=0.841, test=0.478), total=   0.2s\n",
      "[CV] learning_rate=0.1, n_estimators=200 .............................\n",
      "[CV]  learning_rate=0.1, n_estimators=200, score=(train=0.855, test=0.357), total=   0.4s\n",
      "[CV] learning_rate=0.1, n_estimators=200 .............................\n",
      "[CV]  learning_rate=0.1, n_estimators=200, score=(train=0.877, test=0.239), total=   0.3s\n",
      "[CV] learning_rate=0.1, n_estimators=200 .............................\n",
      "[CV]  learning_rate=0.1, n_estimators=200, score=(train=0.823, test=0.511), total=   0.3s\n",
      "[CV] learning_rate=0.1, n_estimators=500 .............................\n",
      "[CV]  learning_rate=0.1, n_estimators=500, score=(train=0.855, test=0.383), total=   0.8s\n",
      "[CV] learning_rate=0.1, n_estimators=500 .............................\n",
      "[CV]  learning_rate=0.1, n_estimators=500, score=(train=0.912, test=0.387), total=   0.7s\n",
      "[CV] learning_rate=0.1, n_estimators=500 .............................\n",
      "[CV]  learning_rate=0.1, n_estimators=500, score=(train=0.823, test=0.468), total=   0.9s\n",
      "[CV] learning_rate=0.01, n_estimators=50 .............................\n",
      "[CV]  learning_rate=0.01, n_estimators=50, score=(train=0.467, test=0.212), total=   0.1s\n",
      "[CV] learning_rate=0.01, n_estimators=50 .............................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\72445\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "C:\\Users\\72445\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "C:\\Users\\72445\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "C:\\Users\\72445\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "C:\\Users\\72445\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "C:\\Users\\72445\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  learning_rate=0.01, n_estimators=50, score=(train=0.428, test=0.190), total=   0.2s\n",
      "[CV] learning_rate=0.01, n_estimators=50 .............................\n",
      "[CV]  learning_rate=0.01, n_estimators=50, score=(train=0.484, test=0.309), total=   0.1s\n",
      "[CV] learning_rate=0.01, n_estimators=100 ............................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\72445\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "C:\\Users\\72445\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  learning_rate=0.01, n_estimators=100, score=(train=0.467, test=0.212), total=   0.2s\n",
      "[CV] learning_rate=0.01, n_estimators=100 ............................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\72445\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "C:\\Users\\72445\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "C:\\Users\\72445\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  learning_rate=0.01, n_estimators=100, score=(train=0.426, test=0.191), total=   0.3s\n",
      "[CV] learning_rate=0.01, n_estimators=100 ............................\n",
      "[CV]  learning_rate=0.01, n_estimators=100, score=(train=0.433, test=0.420), total=   0.1s\n",
      "[CV] learning_rate=0.01, n_estimators=200 ............................\n",
      "[CV]  learning_rate=0.01, n_estimators=200, score=(train=0.645, test=0.217), total=   0.4s\n",
      "[CV] learning_rate=0.01, n_estimators=200 ............................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\72445\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  learning_rate=0.01, n_estimators=200, score=(train=0.629, test=0.175), total=   0.3s\n",
      "[CV] learning_rate=0.01, n_estimators=200 ............................\n",
      "[CV]  learning_rate=0.01, n_estimators=200, score=(train=0.626, test=0.381), total=   0.3s\n",
      "[CV] learning_rate=0.01, n_estimators=500 ............................\n",
      "[CV]  learning_rate=0.01, n_estimators=500, score=(train=0.800, test=0.196), total=   0.6s\n",
      "[CV] learning_rate=0.01, n_estimators=500 ............................\n",
      "[CV]  learning_rate=0.01, n_estimators=500, score=(train=0.817, test=0.269), total=   0.7s\n",
      "[CV] learning_rate=0.01, n_estimators=500 ............................\n",
      "[CV]  learning_rate=0.01, n_estimators=500, score=(train=0.876, test=0.457), total=   0.8s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  36 out of  36 | elapsed:   15.2s finished\n",
      "C:\\Users\\72445\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\model_selection\\_search.py:813: DeprecationWarning: The default of the `iid` parameter will change from True to False in version 0.22 and will be removed in 0.24. This will change numeric results when test-set sizes are unequal.\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=3, error_score='raise-deprecating',\n",
       "             estimator=AdaBoostClassifier(algorithm='SAMME.R',\n",
       "                                          base_estimator=None,\n",
       "                                          learning_rate=1.0, n_estimators=50,\n",
       "                                          random_state=None),\n",
       "             iid='warn', n_jobs=None,\n",
       "             param_grid={'learning_rate': [1, 0.1, 0.01],\n",
       "                         'n_estimators': [50, 100, 200, 500]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "             scoring='f1_macro', verbose=5)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search = GridSearchCV(clf, params, cv=3, return_train_score=True, verbose=5, scoring='f1_macro')\n",
    "\n",
    "search.fit(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuned Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Training Score: 0.7762419456686359\n",
      "Mean Testing Score: 0.8439944283724419\n",
      "\n",
      "Best Parameter Found:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'learning_rate': 1, 'n_estimators': 500}"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Mean Training Score:\", np.mean(search.cv_results_['mean_train_score']))\n",
    "print(\"Mean Testing Score:\", search.score(X, y))\n",
    "print(\"\\nBest Parameter Found:\")\n",
    "search.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model with the Best Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None, learning_rate=1,\n",
       "                   n_estimators=500, random_state=None)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search_clf = search.best_estimator_\n",
    "\n",
    "search_clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Results from Optimum Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        Sell       0.22      1.00      0.36         2\n",
      "         Buy       1.00      0.31      0.47        13\n",
      "        Hold       0.50      1.00      0.67         2\n",
      "\n",
      "    accuracy                           0.47        17\n",
      "   macro avg       0.57      0.77      0.50        17\n",
      "weighted avg       0.85      0.47      0.48        17\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Classifier predictions\n",
    "s_pred = search_clf.predict(X_test)\n",
    "\n",
    "#Printing out results\n",
    "report = classification_report(y_test, s_pred, target_names=['Sell', 'Buy', 'Hold'])\n",
    "print(report)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confusion Matrix for Optimum Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeEAAAGoCAYAAABxHV2qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deZxkdXXw/8/pbaZnX2WZAeYnAgYENwKyqYiAA4g+xAgqKCQ+xJdbjOQhJvrgkhjNgzFGkSQTjYoiGo0LKIsoIAoEBVnisMkmAzPILDBrz3RP9/n9UTXYNN09vda9VfV5v171ourWt24d5jKcOt/tRmYiSZJqr6XoACRJalYmYUmSCmISliSpICZhSZIKYhKWJKkgJmFJkgpiEpaGEBGdEXFZRKyPiG+N4zxviYgfTWRsRYiIKyLibUXHITUSk7DqXkS8OSJuiYhNEbGqmiyOnIBTvwHYBZifmX881pNk5sWZedwExPMMEfHKiMiI+M6A4y+sHr9uhOf5SER8bWftMnNpZn5ljOFKGoRJWHUtIt4PfAb4eyoJc0/gQuB1E3D6vYD7MnP7BJxrsqwGDo+I+f2OvQ24b6K+ICr8f4U0CfyLpboVEbOBjwHvyszvZObmzOzJzMsy8/9U20yJiM9ExMrq4zMRMaX63isj4tGIOCcinqhW0WdV3/socB5warXC/tOBFWNELKlWnG3V12dGxIMRsTEiHoqIt/Q7/vN+nzs8In5Z7eb+ZUQc3u+96yLibyPihup5fhQRC4b5Y+gGvgecVv18K/BG4OIBf1b/HBErImJDRNwaEUdVj78G+Jt+/5539Ivj4xFxA7AFeG712Nur7/9LRHy73/n/ISJ+EhEx4gsoySSsunYYMBX47jBtPgi8DHgR8ELgEOBD/d7fFZgNLAL+FPh8RMzNzA9Tqa6/mZkzMvOLwwUSEdOBzwJLM3MmcDhw+yDt5gE/rLadD3wa+OGASvbNwFnAc4AO4C+H+27gIuCt1efHA8uBlQPa/JLKn8E84OvAtyJiamZeOeDf84X9PnMGcDYwE/jtgPOdAxxU/YFxFJU/u7el++BKo2ISVj2bD6zZSXfxW4CPZeYTmbka+CiV5LJDT/X9nsy8HNgE7DfGePqAF0REZ2auyszlg7Q5EfhNZn41M7dn5iXAPcBr+7X5Umbel5ldwH9SSZ5DyswbgXkRsR+VZHzRIG2+lplrq9/5j8AUdv7v+eXMXF79TM+A820BTqfyI+JrwHsy89GdnE/SACZh1bO1wIId3cFD2J1nVnG/rR57+hwDkvgWYMZoA8nMzcCpwDuAVRHxw4h4/gji2RHTon6vHx9DPF8F3g0czSA9A9Uu97urXeBPUan+h+vmBlgx3JuZ+QvgQSCo/FiQNEomYdWzm4CtwOuHabOSygSrHfbk2V21I7UZmNbv9a7938zMqzLzWGA3KtXtv48gnh0xPTbGmHb4KvBO4PJqlfq0anfxX1EZK56bmXOA9VSSJ8BQXcjDdi1HxLuoVNQrgXPHHrrUvEzCqluZuZ7K5KnPR8TrI2JaRLRHxNKI+H/VZpcAH4qIhdUJTudR6T4di9uBl0fEntVJYX+9442I2CUiTq6ODW+j0q3dO8g5Lgf2rS6raouIU4H9gR+MMSYAMvMh4BVUxsAHmglspzKTui0izgNm9Xv/d8CS0cyAjoh9gb+j0iV9BnBuRAzbbS7p2UzCqmuZ+Wng/VQmW62m0oX6biozhqGSKG4B7gT+B/hV9dhYvutq4JvVc93KMxNnC5XJSiuBdVQS4jsHOcda4KRq27VUKsiTMnPNWGIacO6fZ+ZgVf5VwBVUli39lkrvQf+u5h0bkayNiF/t7Huq3f9fA/4hM+/IzN9QmWH91R0zzyWNTDiZUZKkYlgJS5JUEJOwJEkTJCL2i4jb+z02RMT7hmxvd7QkSROvuoPdY8ChmTlwaSJgJSxJ0mQ5BnhgqAQMMNwmB3VnxuwFOX+XJUWHoQm09omNRYcgaRjbtjxOT/f60u8Z/tKW6bkhB1s1ODr3s205lRUGOyzLzGVDND+NyjLJITVUEp6/yxI++PlfFh2GJtBXPnt90SFIGsYdPzu76BBGZEP28pm2gfvkjN5J2+/bmpkH76xdRHQAJ9NvP4HBNFQSliRpUAHRPgEF+8hvbLoU+FVm/m64Ro4JS5I08d7ETrqiwUpYktQEIoKWttoMXUfENOBY4M921tYkLElqfAHRXpvO3+pNVObvtCEmYUlSMwhqVgmPhmPCkiQVxEpYktT4Jmp29AQzCUuSGl4tJ2aNhklYktT4SloJOyYsSVJBrIQlSY2vpLOjTcKSpIYXQLSWLwnbHS1JUkGshCVJjS+gpYSVsElYktQEgmgxCUuSVHsB0Vq+EdjyRSRJUpOwEpYkNbzAMWFJkooROCYsSVIxopSVsGPCkiQVxEpYktTwIsq5Y5ZJWJLUFKKlfJ2/5YtIkqQmYSUsSWp8zo6WJKko5ZwdbRKWJDW8KGkl7JiwJEkFsRKWJDWFMs6ONglLkhpfSbujTcKSpCZQzolZ5avNJUlqElbCkqSGV9bZ0SZhSVJTcGKWJElFKGklXL6fBZIkNQkrYUlSE4hSVsImYUlSUyhjErY7WpKkglgJS5IaXmWJUvnqTpOwJKkplHHHLJOwJKnxRTknZpWvNpckqUlYCUuSmoJjwpIkFcC9ozUmrdnFHt1Xsrj7KhZuv4VZvQ/QnpvZFnNY13YQD005hXunnkVvdBYdqsbgqEU/4/X7XMq+c+9nevsm1nQt4OZVh/CNe97IY5sWFR2eRsnrWW61SsIRMQf4AvACIIE/ycybBmtrEi65M9buSkdufNbxzlzDop5rWNRzDQd0XcBVs77PhrZ9CohQY5N84JDzee3elz/j6KIZqzhln+/zmiU/4rwbPsxNq15WUHwaHa+nnuGfgSsz8w0R0QFMG6ph+TrI9QwduZHtTOH+KW/ixzMv4ZJ59/Pl+Wv59tzbWD71HSTB3N57OGH98bTlpqLD1Qi9df+Ln/4f9k9+ezRvveKLnPid7/FX13+clZt2ZVp7Fx874qPsMXNFwZFqJLye9SCIlpZxP3b6LRGzgJcDXwTIzO7MfGqo9ibhkls+9Z1cMv9hrpl1MQ9OPZWNrc+lu2Uu69peyA0zL+QX0/8egFl9D3NA14UFR6uRmDt1HWfs/zUAbnjsZZx343k88NTePLVtDj9/7Ajee80/saVnKtPauzj7oC8UHK12xutZJ6pjwuN9AAsi4pZ+j7MHfNNzgdXAlyLitoj4QkRMHyosk3DJ3TDzArpadhny/Ts7z2FrzAdgj+4raxWWxmHpkh8xrX0rAMvufDvwzHGqVZt347IHTgLgFYt/xtyp62odokbB69l01mTmwf0eywa83wa8BPiXzHwxsBn4wFAnMwnXuYw21rdWxoKn9a0sOBqNxBGLbgTgkQ2Luf+p5w3a5toVrwCgtaWPw3a7uWaxafS8nvWiNt3RwKPAo5m540J/m0pSHpRJuAF09v0OgO6YVXAkGol95/4GgLvW/sGQbe5Ztx/b+yp/Pfebd19N4tLYeD3rSMT4HzuRmY8DKyJiv+qhY4C7hmrv7Og6N7/nNmb1PQTAE22HFhyNdmZB52qmtXcBsHLT7kO26+nrYG3XfHaZvpq9Zj1Sq/A0Sl7P+lHjdcLvAS6uzox+EDhrqIYm4Tp36OZzAUiCuzsHzg9Q2cyZsv7p509tmz1s2ye3zWWX6auZ1bFhssPSGHk9NZjMvB04eCRta9odHREfjIjlEXFnRNweEUOWbhHx5Yh4Q/X5dRExon+hZnLQlvNZ3PMTAO6a+g6ebDuw4Ii0M1Pbtj79vLu3Y9i226rvd7Z1TWpMGjuvZ32p0ZjwqNSsEo6Iw4CTgJdk5raIWAAM/1+thrS4+yoO2fw3AKxtPZD/nvGpgiPSSAT59PNk+K6xGOSZysXrWUdKehelWnZH70Zlavc2gMxcAxARLwU+DcwA1gBnZuaqGsZVdxb03MqrN7yRFnrZ1LIHV87+gdtW1omu7b+/TlNatw3btqO1u/qZqZMak8bO61lfyngDh1pG9CNgj4i4LyIujIhXREQ78DngDZn5UuA/gI+P5qQRcfaORdOb1q+ehLDLZfb2+1i6/gQ6ciNdsZAfzr6Kza17FB2WRqj/uGH/8cTBzJlS2WRnQ7ez3svK66nxqlklnJmbqlXvUcDRwDeBv6OywfXVUZn63QqMqgquLpReBrDXvgfnTprXtem9Kzhh/XF05mq6YxaXz76C9W3PLzosjcKaroVs6elkWnsXu80Y+j/19pZuFnSuAeC3G/asVXgaJa9nfWn27mgysxe4DrguIv4HeBewPDMPq2Uc9Whq32pOXH8cM/seYTudXDnrUta2D7n+WyV235P78KLn3MkB8+8ess1+8+6jraUPgHvX7Vur0DQGXs/6UNZbGdasOzoi9ouI/rf5eRFwN7CwOmmLiGiPiANqFVO9aO/bwNL1S5nTey+9tHP1rG/xeMfLiw5LY3TDY4cDsOesFTx39oODtjl6j58C0NvXwk2rXP9dZl7PehHQ0jL+xwSr5ZjwDOArEXFXRNwJ7A+cB7wB+IeIuAO4HTi8hjGVXmtu5fgNJ7Nw+6/oo4VrZ17EiiknFB2WxuGKh497enLOnw2yof+u01dx8t6XAfDTR4/iya3zahqfRsfrqfGo5ZjwrQyeYNdQue3TwPZn9nv+ykkLrMQiezlmw2ns3nM9ADdPP59Hppw45C0LkxZ6Y8jbVqokntw6j4uWn86fvfALHLn4Rj56+Ef5yvIzWLd1HvvPv4v3vfRzTGvfypaezuoNAVRmXs/6ESPYdrLW3DGrxKb3rWBJ96VPvz5s8zkctvmcIdtvbNmLS+Y/VIvQNE4X3fUWdp+xktfufTmv3utaXr3Xtc94f0tPJ+fd8GFWbHTmez3wetaBKOcSJZOwVIjgk784lxtXHsbrnncp+839DdPaN7OmawG/WPWHXHLPqTy2aVHRQWrEvJ4aG5NwiW1qXcKyhX1Fh6FJdP2jR3H9o0cVHYYmiNezzNwxS5KkYgSTMrt5vEzCkqSmUMZKuHw/CyRJahJWwpKkhhcEEeWrO03CkqTGF0AJu6NNwpKkplDGdcLli0iSpCZhJSxJagplnB1tEpYkNb7KvQyLjuJZyheRJElNwkpYktQU7I6WJKkoJZwdbRKWJDW8iCjl/YTL97NAkqQmYSUsSWoOdkdLklQMJ2ZJklQE1wlLkqT+rIQlSc3B7mhJkorh/YQlSSpCSe8nXL6fBZIkNQkrYUlSEwjCdcKSJBXEbSslSdIOVsKSpMYXuG2lJEnFiFJ2R5uEJUlNoYwTs8oXkSRJTcJKWJLU+IKa3cAhIh4GNgK9wPbMPHiotiZhSVITiFrvmHV0Zq7ZWSOTsCSp4QXl3Du6fBFJklReCyLiln6Pswdpk8CPIuLWId5/mpWwJKnxTdwNHNYMN8ZbdURmroyI5wBXR8Q9mXn9YA2thCVJTSAqE7PG+xiBzFxZ/ecTwHeBQ4ZqaxKWJGmCRMT0iJi54zlwHPDrodrbHS1Jag612TFrF+C7UfmuNuDrmXnlUI1NwpKk5lCDHbMy80HghSNtbxKWJDW+iJpt1jEa5YtIkqQmYSUsSWoOtd0xa0RMwpKk5lDC7miTsCSpOZTwfsLl+1kgSVKTsBKWJDW+iJosURotk7AkqTmUsDvaJCxJag4lnJhVvogkSWoSVsKSpMbnmPDk6/jNr9l96fOLDkMT6LJb/73oEDTBvv3EUUWHoAn08XtnFh3CyJVwTLh8PwskSWoSDVUJS5I0pBJOzDIJS5KaQJSyO9okLElqfEEpJ2aVLyJJkpqElbAkqeElkHZHS5JUhHBiliRJhSlhEi5fRJIkNQkrYUlSU3BMWJKkIkQ5x4TLF5EkSU3CSliS1BzsjpYkqSAl3DHLJCxJagJRyolZ5ftZIElSk7ASliQ1vqCUs6NNwpKkppAmYUmSilDO+wmX72eBJElNwkpYktQU7I6WJKkoJeyONglLkhqfe0dLkqT+rIQlSQ0v8VaGkiQVx+5oSZK0g5WwJKkpJHZHS5JUgHCdsCRJhalREo6IVuAW4LHMPGm4tuX7WSBJUn37c+DukTQ0CUuSGl9UliiN97HTr4lYDJwIfGEkYdkdLUlqeFm7MeHPAOcCM0fS2EpYktQcIsb/gAURcUu/x9m/P32cBDyRmbeONCQrYUmSRm5NZh48xHtHACdHxAnAVGBWRHwtM08f6mRWwpKkppDRMu7HsOfP/OvMXJyZS4DTgGuGS8BgJSxJagrhZh2SJBWllpt1ZOZ1wHU7a2d3tCRJBbESliQ1vmDH7OZSGVMlHBHvm+hAJEmaPEHSMu7HRBvrGd8/oVFIktSExtodXb6aXpKkISSMaNvJWhtrEs4JjUKSpElWV7cyjIiNDJ5sA5g2aRFJkjQJ6mqdcGaOaPNpTZ5X3XMV0/ZaNOL2t//vD/Lo174/iRFpsrRse5TZD5xFZDcAm3b/AN1zlxYclUaiNbvYo/tKFndfxcLttzCr9wHaczPbYg7r2g7ioSmncO/Us+iNzqJDVQmNqjs6IqYDrwfenJknTk5IGquNy39TdAgao+mrPv10AlZ9OWPtrnTkxmcd78w1LOq5hkU913BA1wVcNev7bGjbp4AIVVGzuyiNyk6TcER0ACcAbwZeA/wX8K+THJeA6178OqJlmO6TCF55+6V0LtqVjXc/wPrb7qpdcJowHU/9iPbNt9LbvhutPauKDkej1JEb2c4UHp5yCg93nMzq9j9kW8xlRt8j/EHXv7H/1n9jbu89nLD+eL497062x4yiQ25adTUxKyKOBd4EHA9cC3wVOCQzz6pRbE2vr2vrsO/PO/JgOhftCsCjX7+sFiFpgkXvRqY9fiFJK1t2fTczV3yw6JA0SsunvpNfTf+/dLXs8ozj61rmcsPMC9nUuheHbv5rZvU9zAFdF3LHtHMLirS5JeUcEx6uNr8K2Bs4MjNPz8zLgL7ahKWRWPzm1wKQfX089o0fFByNxqLz8X+hpfdJts07hd6pexcdjsbghpkXPCsB93dn5zlsjfkA7NF9Za3CUp0Yrjv6pVRuxfTjiHgQ+AbQWpOotFMtUzrY7X8dC8Da63/J1kcfLzgijVbb5juZ8tTl9LXNZ8tzzqKld0PRIWkSZLSxvnUfpm5fy7S+lUWH07yinGPCQ0aUmbdl5l9l5t7AR4AXAx0RcUVEnF2rADW4XU46mvY5swB49OJLC45Go5bbmbbqHwmSLbu+C1qnFx2RJlFn3+8A6I5ZBUfS3LJ6O8PxPCbaiH4WZOYNmfluYBHwGeCwCY9Eo7L4TZWu6O2bt7Dqe1cXHI1Ga+qaS2jb9jA9019K9+xjig5Hk2h+z23M6nsIgCfaDi04GpXNcBOzXjLgUAJrMvMqKuPFKkj7/DksPPYIAH532TX0btpScEQajZbux+hcfREZ7WzezXuhNLpDN1cmYiXB3Z12IhapjN3Rw40J/+Mgx+ZVlyydlpl3jOaLIqIX+B8qO271Au/OzBtHcw5VLHrjCbR0tAPOiq5H01dW1gR3LTidvil7Fh2OJtFBW85ncc9PALhr6jt4su3AgiNqbmWcHT3cjllHD3Y8Ig4GPge8fJTf1ZWZL6qe43jgE8ArRnkOAYvedBIAW1etZvVPbio4Go1Gx1NX0775Fnrbd6Vr4VuLDkeTaHH3VRyy+W8AWNt6IP8941MFR9TcsqSbdYw6osy8BRjvavNZwJMAEfHKiHh6fU1EXBARZ0bEMRHx3X7Hj42I74zze+ve9Oftxdw/PAiAx/7zh9DnqrF6Eb0bmfa7CwHYstt7oWVKwRFpsizouZVXb3gjLfSyqWUPrpz9A7et1KBGfReliNiFsd1FqTMibgemArsBr9pJ+2uAz0fEwsxcDZwFfGkM39tQFlXXBgM8Zld0Xel84iu0bF9H98wj6Jl5RNHhaJLM3n4fS9efQEdupCsW8sPZV7G5dY+iwxJ11h0dEZ/j2cl2HnA48Odj+K7+3dGHARdFxAuGapyZGRFfBU6PiC9RmZH9rP676nKpswEWjvnOjPVjcbUresOv72PDnfcWHI1Go6W6JWXHxhuYt3z4kZgZKz8JKz8JwFP7fIO+jt0mPT6N3/TeFZyw/jg6czXdMYvLZ1/B+rbnFx2Wqupq20rglgGvE1gLvD8znxjPl2bmTRGxAFgIbOeZ3eJT+z3/EnAZsBX4VmZuH+Rcy4BlAPvE1Ia+z/G8I17CtCWLASdkSWUztW81J64/jpl9j7CdTq6cdSlr2wcuMlGRMusrCR+dmWdOxpdGxPOp7L61FvgtsH9ETKGSgI8Bfg6QmSsjYiXwIeDYyYilniyqrg3O3l5WfvOHBUej0dqy67vpWnjmkO+3bF/LzEf+qtJ24VlPd1n3tS2oRXgah/a+DSxdv5Q5vffSSztXz/oWj3eMdu6qmtFwSfigCf6uHWPCUFmm9LbM7AVWRMR/AncCvwFuG/C5i4GFmdnUtwhq6Whnt1OOA2DNtTezdeW4OiNUgJ11KWf37+c79rXvQm+nt72rB625leM3nMzC7b+ijxaunXkRK6acUHRYepYgRz8XedINl4SnRcSLYfCR7Mz81Wi+KDOH3Hc6M88Fhrq1yJHAv4/muxrRLie+ko65swG7oqWyiOzlmA2nsXvP9QDcPP18HplyIm25adD2SQu9Ma2WIaqqrHdRGi4JL6KyYcdgUSc7n908bhFxK7AZOGeyv6vsdnRFb9+0hce//+OCo5EEML1vBUu6f793+2Gbz+GwzUP/72pjy15cMv+hWoSmQdRbEr4/Myc90Q4nM19a5PeXRfu82Tzn+KMAePz7P6Z3S1fBEUmSJkLjr+lpAD3r1nP57BcXHYYmWV/Hbqw74KdFh6FR2NS6hGUL3TCnXpSxEh5ulPoTEbH/wIMRcUBELJzEmCRJmmDjv41hrW9leAqVdbwDLQb+ecIjkSSpyQyXhA/MzGf1jVVvZTjRy5ckSZpUmTHux0Qbbky4Y5j32ic6EEmSJktZlygNVwnfFxHPWnEeEUuBBycvJEmSJl4Zx4SHq4T/AvhBRLwRuLV67GAqN1I4acIjkSSpyQxZCWfmfcCBwE+BJcBewHXAnzC2uyhJklSYequEycxtwJeq21e+Cfgw8BDwXxMeiSRJk2ZyJlaN13D3E94XOI1K8l0LfBOIzDy6RrFJkjQhEugr4cSs4Srhe4CfAa/NzPsBIuIvahKVJElNYLjZ0X8EPA5cGxH/HhHHMMQdlSRJKrsyjgkPNzHru5l5KvB8KhOy/gLYJSL+JSKOm/BIJEmaLFnOzTp2eofjzNycmRdn5klUtqy8HfjAhEciSVKT2WkS7i8z12XmvxV9i0NJkkarFt3RETE1In4REXdExPKI+Ohw7b2VoSSpCdRsidI24FWZuSki2oGfR8QVmfnfgzU2CUuSGl6t9o7OzAQ2VV+2Vx85VPtRdUdLkqThRURrRNwOPAFcnZk3D9XWJCxJagoTNDt6QUTc0u9x9rO/J3sz80VUJjMfEhEvGComu6MlSU2hb2JOsyYzDx5Jw8x8KiKuA14D/HqwNlbCkqSmUIt1whGxMCLmVJ93Aq+msgPloKyEJUmaOLsBX4mIViqF7n9m5g+GamwSliQ1vMnadvJZ35N5J/DikbY3CUuSmkJd3cpQkqRGUotKeLScmCVJUkGshCVJjS+hb8h9q4pjEpYkNbxabVs5WnZHS5JUECthSVJTcHa0JEkFSceEJUkqQtDnmLAkSdrBSliS1PASx4QlSSqMY8KSJBXEdcKSJOlpVsKSpMbntpWSJBWjrBOz7I6WJKkgVsKSpKbg7GhJkgpSxh2zTMKSpKZQxkrYMWFJkgpiJSxJanhJlHJ2tElYktT4XCcsSVJxHBOWJElPsxKWJDWFMt7AwSQsSWp4iWPCkiQVpoxjwg2VhFfN3otPHLWs6DA0kT5Uwr81Gpe3vbfoCKTyaKgkLEnSUKyEJUkqQCb0lXCzDpcoSZJUECthSVJTsDtakqSCmIQlSSpIGdcJOyYsSVJBrIQlSQ0vwVsZSpJUiHRMWJKkwjgmLEmSnmYlLElqeJUx4aKjeDYrYUlSU8gc/2NnImKPiLg2Iu6OiOUR8efDtbcSliRp4mwHzsnMX0XETODWiLg6M+8arLFJWJLUFGoxMSszVwGrqs83RsTdwCLAJCxJalIFLFGKiCXAi4Gbh2pjEpYkNbwE+vom5FQLIuKWfq+XZeaygY0iYgbwX8D7MnPDUCczCUuSNHJrMvPg4RpERDuVBHxxZn5nuLYmYUlSU6hFd3REBPBF4O7M/PTO2rtESZLUFGqxRAk4AjgDeFVE3F59nDBUYythSVLDy6zZ7OifAyO+U4SVsCRJBbESliQ1hSzhvpUmYUlSUyhhDjYJS5KawwStE55QjglLklQQK2FJUsMbxRKjmjIJS5KaQi2WKI2W3dGSJBXESliS1BTsjpYkqSBZwv5ok7AkqeHVatvK0XJMWJKkglgJS5KagmPCkiQVpK+E/dEmYUlSw0vKWQk7JixJUkGshCVJjc9tKyVJKkrSV8IsbHe0JEkFsRKWJDWFLOH9hE3CkqSGV5kdXb7uaJOwJKnxJfSVsBJ2TFiSpIJYCUuSmoLd0RqXoxb9jNfvcyn7zr2f6e2bWNO1gJtXHcI37nkjj21aVHR4GgOvaf1rzS726L6Sxd1XsXD7LczqfYD23My2mMO6toN4aMop3Dv1LHqjs+hQm1pSzrsomYTrQvKBQ87ntXtf/oyji2as4pR9vs9rlvyI8274MDetellB8Wn0vKaN4oy1u9KRG591vDPXsKjnGhb1XMMBXRdw1azvs6FtnwIiFFDZrKOEWdgx4Trw1v0vfvp/1j/57dG89YovcuJ3vsdfXf9xVm7alWntXXzsiI+yx8wVBUeqkfKaNo6O3Mh2pnD/lDfx45mXcMm8+/ny/LV8e+5tLJ/6DpJgbu89nLD+eNpyU9HhqmRMwiU3d+o6ztj/awDc8NjLOO/G83jgqb15atscfv7YEbz3mn9iS89UprV3cfZBXyg4Wo2E17SxLJ/6Ti6Z/zDXzEFPlSMAABAfSURBVLqYB6eeysbW59LdMpd1bS/khpkX8ovpfw/ArL6HOaDrwoKjbW6Z439MNJNwyS1d8iOmtW8FYNmdbwfiGe+v2rwblz1wEgCvWPwz5k5dV+sQNUpe08Zyw8wL6GrZZcj37+w8h60xH4A9uq+sVVgaRF9fjvsx0UzCJXfEohsBeGTDYu5/6nmDtrl2xSsAaG3p47Ddbq5ZbBobr2lzyWhjfWtlLHha38qCo2lemTkhj4lmEi65fef+BoC71v7BkG3uWbcf2/sql3K/effVJC6Nnde0+XT2/Q6A7phVcCQqG5NwiS3oXM209i4AVm7afch2PX0drO2qdHftNeuRmsSmsfGaNp/5Pbcxq+8hAJ5oO7TgaJpb9o3/MdFMwiU2Z8r6p58/tW32sG2f3DYXgFkdGyY1Jo2P17T5HLr5XACS4O7OswuOprn1ZY77MdEmLQlHxKYBr8+MiAt28pmPRMRfDnJ8SUT8eqJjLLupbVufft7d2zFs223V9zvbuiY1Jo2P17S5HLTlfBb3/ASAu6a+gyfbDiw4IpWNm3WUWPD7X105YAbts9s++5nKx2vaPBZ3X8Uhm/8GgLWtB/LfMz5VcERy28qqiNgL+A9gIbAaOCszHxnQ5qXVNluAn9c8yBLo2v77be6mtG4btm1Ha3f1M1MnNSaNj9e0OSzouZVXb3gjLfSyqWUPrpz9A7etLFgmk7LEaLwmc0y4MyJu3/EAPtbvvQuAizLzIOBi4LODfP5LwHsz87DhviQizo6IWyLilp7u9cM1rTv9xwz7jyUOZs6UpwDY0O3syzLzmja+2dvvY+n6E+jIjXTFQn44+yo2t+5RdFii+Tbr6MrMF+14AOf1e+8w4OvV518Fjuz/wYiYDczJzJ/2azOozFyWmQdn5sHtHcNPdKk3a7oWsqWn8ut5txmrhmzX3tLNgs41APx2w541iU1j4zVtbNN7V3DC+uPozNV0xywun30F69ueX3RYKrGyzI4e+PsiBjnWlO57srLI/4D5dw/ZZr9599HWUpk7f++6fWsSl8bOa9qYpvat5sT1xzGz7xG208mVsy5lbftLig5L/WRfjvsx0YpKwjcCp1Wfv4UBY76Z+RSwPiKO7NemKd3w2OEA7DlrBc+d/eCgbY7eo9Jh0NvXwk2rXIdYdl7TxtPet4Gl65cyp/deemnn6lnf4vGOlxcdlvrJCVieVFdLlHbivcBZEXEncAbw54O0OQv4fETcBDTtGo0rHj7u6Yk5fzbIZv67Tl/FyXtfBsBPHz2KJ7fOq2l8Gj2vaWNpza0cv+FkFm7/FX20cO3Mi1gx5YSiw9IgmqoSzswZA15/OTPfXX3+cGa+KjMPysxjdsyMzsyPZOanqs9vzcwXZuZh1eMvmKxYy+zJrfO4aPnpABy5+EY+evhHee7sB5kz5SkO3/1GPvuq9zOtfStbejqrNwNQ2XlNG0dkL8dsOI3de64H4Obp5/PIlBNpy02DPlpzS8ERa7JFxH9ExBMj3dvCdcJ14KK73sLuM1by2r0v59V7Xcur97r2Ge9v6enkvBs+zIqNzsCsF17TxjC9bwVLui99+vVhm8/hsM3nDNl+Y8teXDL/oVqEpkFMRiU7iC9TXQE0ksYm4boQfPIX53LjysN43fMuZb+5v2Fa+2bWdC3gF6v+kEvuOZXHNi0qOkiNitdUqqmEWuTgzLw+IpaMtL1JuI5c/+hRXP/oUUWHoQnkNa1vm1qXsGzhJOzqr6ZhEpYkNbxkwrqjF0TELf1eL8vMZWM9mUlYktQEcqL2jl6TmQdPxInAJCxJagZNuHe0JElNJSIuAW4C9ouIRyPiT4drbyUsSWoKtbiVYWa+aTTtTcKSpIY3gROzJpRJWJLU+LKcSdgxYUmSCmIlLElqApNzF6TxMglLkppCGbujTcKSpIaX1GZ29Gg5JixJUkGshCVJja+kO2aZhCVJTaGMY8J2R0uSVBArYUlSE5iwuyhNKJOwJKnhZUL29RUdxrOYhCVJTaGME7McE5YkqSBWwpKkpuCYsCRJRcgs5RIlk7AkqeGV9X7CjglLklQQK2FJUlPoS5coSZJUe2l3tCRJ6sdKWJLU8BJnR0uSVBjXCUuSVISEvhLuHe2YsCRJBbESliQ1BceEJUkqQJKk64QlSSqA64QlSVJ/VsKSpKZQxkrYJCxJagLp3tGSJBUhHROWJEn9WQlLkppClnDHLJOwJKnx2R0tSZL6sxKWJDUBd8ySJKkQCfSVsDvaJCxJanxZzolZjglLklQQK2FJUhNIZ0dLklSUzL5xP3YmIl4TEfdGxP0R8YGdtbcSliQ1vhqsE46IVuDzwLHAo8AvI+LSzLxrqM9YCUuSNDEOAe7PzAczsxv4BvC64T5gJSxJanhJ1mJ29CJgRb/XjwKHDveByCzfQPVYRcRq4LdFx1EDC4A1RQehCeU1bTzNck33ysyFRQexMxFxJZVrMl5Tga39Xi/LzGXV7/hj4PjMfHv19RnAIZn5nqFO1lCVcD38hzARIuKWzDy46Dg0cbymjcdrWi6Z+ZoafM2jwB79Xi8GVg73AceEJUmaGL8E9omI/y8iOoDTgEuH+0BDVcKSJBUlM7dHxLuBq4BW4D8yc/lwnzEJ16dlRQegCec1bTxe0yaUmZcDl4+0fUNNzJIkqZ44JixJUkFMwpIkFcQkXGciIoqOQdLQImJO0TGofpiE60hE/CHw1ojoLDoWTbyIcKJknYuIRcANEfGqomNRfTAJ15fpwLuBP4qIqUUHo4kTEfsC/xIRU4qORWMTEZGZjwHnA+dHxMuKjknlZxKuAxFxYESckZnXAecAbwfeaCKuf/2GFzqAPiprC1Vnqgl4x1KTh6lsn7ssIg4vLirVA5NwfTgQ+F8R8ebMvB74CPAnmIgbwazqP+8FdgE+XGAsGqMdCbi6UcPHqdw95xrgXyPi5UXGpnIzCZfYjiopM78OfAs4MSJOr1bEH6GSiN/gGHF9iojFwEUR8aeZ2UNlqGF6ROxVcGgaoYjYLyKW9ju0D/B/M/M/gfcD/wr8U0QcWUiAKj0ngpTUgO4tMvOSiNgAnBERZObXIuI84LNAD/DNomLV6EXEnlRue/Zp4C8j4iAqlXAn8HzgtwP/G1C5REQ78EfAouqlupLK9TsduCYz+yLiGuBNwCcj4tjM7CowZJWQO2aVXET8b2BPKrfO+jxwJJVNwS/PzK9HxBHAo5nZDLdwrHsR0QLMBj5J5b6j/wgEMIfKeP+rgY3AGzLz8aLi1MhExK7AW4HdqPwQvpvKloU3Z+b7I+JU4AXABZn5u+IiVVmZhEsmIqZl5pbq8/cCJwMfAz4D/Fdmfrx6z8q3Al+tdnup5AZWtdUlLH8MPAh8NzPvrx4/APgz4IuZeUchwWpYg1zLhVSGhvagMhb8G+C/qNzC7kAqP6iG3cRfzcsx4RKJiBOAv4+IPSKilcpf6uOBg4HHqSx76MjMbwH/BtxQXLQajczMiHhhRHyu+voa4BIqY4inRsRzq8eXU7kH6XGFBash9U/AEfHaiHgNsF9m/gOVGdGnAntm5pHAmcCRJmANxyRcEhFxEvAJ4LrMXEFlucpi4DoqXdCvy8xu4E8i4uTM/EF1TaJKKiL2johTIuL11UM9wLyI+Kfq/8yvB34IvAM4JSLmRMR0Kl3TI74Li2ovIt5JpYfqSODfI+KDmXk+leVJ74iIYzJzS2auLTJOlZ9JuASq40rnAG/PzO9FxNTqr+0vUxlr+lpm9kTEmcCfA/6yLrnq5hvfB44Azo2IP8nMu6gsX5lNZXgB4A7gNuDKzHwqMzcDS62eyiUinhcRs6s9Gs+hMpTw5sz8EHA4lR/HZwJfAH4N/E9x0aqeODu6HLZRqZK2Vtf9fiAiXkFlgs46Kov+lwIvAv4oMx8oLlTtTETsD1wM/HVmXhYRpwOzIuKAzFweEf8P+HhE3ESl6n1fZv66X1dnd4Hha4CImAu8C+iOiE9k5hMRsZbqdcrMJyPiL4AjMvPLEfHZzOwtMmbVDydmlUB1PfD7qYwDHgD8GPg5cBfweuA+4LtAS2auLipOjUx1Tej1mdlSfX0n8BiwO3BbZp5ZPX4i8Fhm3l5UrBrajh9F1b+fr6FS8fYCHwX+HjgWeFlmbo+I9wAvozJhss+lZRopK+ESqP5F/zfgRiqTsb6fmdsAIuJs4E7HlupHZv48Ik6MiAepzH7+dmZ+LCI6gP+JiA9l5t9l5g8LDlXDawW2UylWroiIWcC5wObM/OuImAlcX/2RdSjwFitgjZaVcIlVlyJ9AHijXdD1JyKOAa4COjKzr3rsT4E5mfmPhQanYUXEAuAW4JBq9/PuVHatuwPYBDyZmZ+IiJdQGeN/ODMfKi5i1SsnZpVQROwWEe+jsjXl20zA9Skzf0Jlnfd9UJncA/wfnLRTepm5BngPcE1EvAD4KvD1zHwnlZnrz4mIfwDuz8xrTcAaK7ujy+kpKgv+X7djEwfVp8y8PCL6ImIL8BCVSVg/Kjou7Vx1Ul0PcCfwN5n5+epbPwOmAEdV/ymNmd3RUg1Uu6ZnZeZ3i45FoxMRxwKfAw7NzPX9jj+9u500ViZhqYa8KUN9qi4R/AxwWGauKzoeNQ67o6UaMgHXp+rs6A7gxxFxcOWQ11LjZyUsSSMUETMyc1PRcahxmIQlSSqIS5QkSSqISViSpIKYhCVJKohJWJKkgrhESRpCRPRS2WKyDbibyhaiWwYcfwg4IzOfiogl1Xb39jvNpzPzooh4mMqtKaFyY4DvAH+bmduqn/tBZr6g+r2HAJ8CdgGSyh21bgP+d/Xz+1e/oxe4ErgHOJ/KnZp2eDOwpRrPPcDU6vd/PjO/Ms4/GkkTxNnR0hAiYlNmzqg+vxi4NTM/PeD4V4D7MvPjA5PpgHM9DBycmWsiYgawDOjJzLf1/1xE7AL8AjgtM2+q3kbvj4CfZebvBp6r+vrM6ut3D/jOZ8QTEc+lkvz/OTO/NEF/TJLGwe5oaWR+BjxvkOM3AYtGc6LqOtN3AK+PiHkD3n4X8JXMvKnaNjPz2zsS8Hhk5oNU7lv93vGeS9LEMAlLOxERbcBSBtz9KCJagWOAS/sd3jsibu/3OGqwc2bmBipd2fsMeOsFwK1jCPPUAd/bOUS7XwHPH8P5JU0Cx4SloXVGxO3V5z8Dvjjg+BIqCfPqfp95IDNfNMLzx4REWfHNQbqjJ/s7JY2TlbA0tK7MfFH18Z7M7O5/HNgL6KDShTwqETGTShK/b8Bby4GXjiPmnXkxlclakkrAJCyNUfW2du8F/jIi2kf6uerErAuB72XmkwPevgB4W0Qc2q/96RGx63jjrU7U+hSV2/JJKgG7o6VxyMzbIuIO4DQqXdZ79+vCBviPzPxs9fm11dnOLcB3gb8d5Hy/i4jTgE9FxHOAPuB6KrOah3NqRBzZ7/U7gZXVeG7j90uUPufMaKk8XKIkSVJB7I6WJKkgJmFJkgpiEpYkqSAmYUmSCmISliSpICZhSZIKYhKWJKkg/z8tqxtitA+OjwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_confusion_matrix(y_test, s_pred, title=\"Confusion Matrix\")\n",
    "np.set_printoptions(precision=1)\n",
    "# Plot non-normalized confusion matrix\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save the optimal model we trained"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['../save_models/adaboost_model_corr.m']"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(search_clf, \"../save_models/adaboost_model_corr.m\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
